{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bT0to3TL2q7H"
   },
   "source": [
    "# Ungraded Lab: Transfer Learning\n",
    "\n",
    "In this lab, you will see how you can use a pre-trained model to achieve good results even with a small training dataset. This is called _transfer learning_ and you do this by leveraging the trained layers of an existing model and adding your own layers to fit your application. For example, you can:\n",
    "\n",
    "1. just get the convolution layers of one model\n",
    "2. attach some dense layers onto it\n",
    "3. train just the dense network\n",
    "4. evaluate the results\n",
    "\n",
    "Doing this will allow you to save time building your application because you will essentially skip weeks of training time of very deep networks. You will just use the features it has learned and tweak it for your dataset. Let's see how these are done in the next sections."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-12slkPL6_JH"
   },
   "source": [
    "## Setup the pretrained model\n",
    "\n",
    "You will need to prepare the pretrained model and configure the layers that you need. For this exercise, you will use the convolution layers of the [InceptionV3](https://arxiv.org/abs/1512.00567) architecture as your base model. To do that, you need to:\n",
    "\n",
    "1. Set the input shape to fit your application. In this case. set it to `150x150x3` as you've been doing in the last few labs.\n",
    "\n",
    "2. Pick and freeze the convolution layers to take advantage of the features it has learned already.\n",
    "\n",
    "3. Add dense layers which you will train.\n",
    "\n",
    "Let's see how to do these in the next cells."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3VqhFEK2Y-PK"
   },
   "source": [
    "First, in preparing the input to the model, you want to fetch the pretrained weights of the `InceptionV3` model and remove the fully connected layer at the end because you will be replacing it later. You will also specify the input shape that your model will accept. Lastly, you want to freeze the weights of these layers because they have been trained already."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "1xJZ5glPPCRz"
   },
   "outputs": [],
   "source": [
    "# Download the pre-trained weights. No top means it excludes the fully connected layer it uses for classification.\n",
    "# The weights have been pre-downloaded for you from the following URL:\n",
    "# https://storage.googleapis.com/mledu-datasets/inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KsiBCpQ1VvPp"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Set the weights file you downloaded into a variable\n",
    "local_weights_file = '/tf/model/inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5'\n",
    "\n",
    "# Initialize the base model.\n",
    "# Set the input shape and remove the dense layers.\n",
    "pre_trained_model = tf.keras.applications.inception_v3.InceptionV3(\n",
    "    input_shape = (150, 150, 3),\n",
    "    include_top = False,\n",
    "    weights = None)\n",
    "\n",
    "# Load the pre-trained weights you downloaded.\n",
    "pre_trained_model.load_weights(local_weights_file)\n",
    "\n",
    "# Freeze the weights of the layers.\n",
    "for layer in pre_trained_model.layers:\n",
    "    layer.trainable = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1y2rEnqFaa9k"
   },
   "source": [
    "You can see the summary of the model below. You can see that it is a very deep network. You can then select up to which point of the network you want to use. As Laurence showed in the exercise, you will use up to `mixed7` as your base model and add to that. This is because the original last layer might be too specialized in what it has learned so it might not translate well into your application. `mixed7` on the other hand will be more generalized and you can start with that for your application. After the exercise, feel free to modify and use other layers to see what the results you get."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qeGP0Ust5kCR",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pre_trained_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jDmGO9tg5iPc"
   },
   "outputs": [],
   "source": [
    "# Choose `mixed7` as the last layer of your base model\n",
    "last_layer = pre_trained_model.get_layer('mixed7')\n",
    "print('last layer output shape: ', last_layer.output.shape)\n",
    "last_output = last_layer.output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UXT9SDMK7Ioa"
   },
   "source": [
    "## Add dense layers for your classifier\n",
    "\n",
    "Next, you will add dense layers to your model. These will be the layers that you will train and is tasked with recognizing cats and dogs. You will add a [Dropout](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dropout) layer as well to regularize the output and avoid overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BMXb913pbvFg",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Flatten the output layer to 1 dimension\n",
    "x = tf.keras.layers.Flatten()(last_output)\n",
    "# Add a fully connected layer with 1,024 hidden units and ReLU activation\n",
    "x = tf.keras.layers.Dense(1024, activation='relu')(x)\n",
    "# Add a dropout rate of 0.2\n",
    "x = tf.keras.layers.Dropout(0.2)(x)\n",
    "# Add a final sigmoid layer for classification\n",
    "x = tf.keras.layers.Dense  (1, activation='sigmoid')(x)\n",
    "\n",
    "# Append the dense network to the base model\n",
    "model = tf.keras.Model(pre_trained_model.input, x)\n",
    "\n",
    "# Print the model summary. See your dense network connected at the end.\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aYLGw_RO7Z_X"
   },
   "source": [
    "## Prepare the dataset\n",
    "\n",
    "Now you will prepare the dataset. This is basically the same code as the one you used in the data augmentation lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "WOV8jON3c3Jv"
   },
   "outputs": [],
   "source": [
    "BASE_DIR = '/tf/cats_and_dogs_filtered'\n",
    "\n",
    "train_dir = os.path.join(BASE_DIR, 'train')\n",
    "validation_dir = os.path.join(BASE_DIR, 'validation')\n",
    "\n",
    "# Directory with training cat/dog pictures\n",
    "train_cats_dir = os.path.join(train_dir, 'cats')\n",
    "train_dogs_dir = os.path.join(train_dir, 'dogs')\n",
    "\n",
    "# Directory with validation cat/dog pictures\n",
    "validation_cats_dir = os.path.join(validation_dir, 'cats')\n",
    "validation_dogs_dir = os.path.join(validation_dir, 'dogs')\n",
    "\n",
    "# Prepare the training set\n",
    "train_dataset = tf.keras.utils.image_dataset_from_directory(\n",
    "    train_dir,\n",
    "    image_size=(150, 150),\n",
    "    batch_size=20,\n",
    "    label_mode='binary'\n",
    "    )\n",
    "\n",
    "# Prepare the validation set\n",
    "validation_dataset = tf.keras.utils.image_dataset_from_directory(\n",
    "    validation_dir,\n",
    "    image_size=(150, 150),\n",
    "    batch_size=20,\n",
    "    label_mode='binary'\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using pretrained models, you should make it a habit to check the documentation for any preprocessing steps. In this case, the [InceptionV3 documentation](https://www.tensorflow.org/api_docs/python/tf/keras/applications/inception_v3/InceptionV3) says that the inputs should be scaled to the range [-1,1]. It has a [`preprocess_input()`](https://www.tensorflow.org/api_docs/python/tf/keras/applications/inception_v3/preprocess_input)  method that you can use to rescale the inputs. The cell below defines a `preprocess` function that uses this method, which you can then map to the datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the preprocess function\n",
    "def preprocess(image, label):\n",
    "    image = tf.keras.applications.inception_v3.preprocess_input(image)\n",
    "    return image, label\n",
    "\n",
    "# Apply the preprocessing to the datasets\n",
    "train_dataset_scaled = train_dataset.map(preprocess)\n",
    "validation_dataset_scaled = validation_dataset.map(preprocess)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optimize the datasets for training\n",
    "SHUFFLE_BUFFER_SIZE = 1000\n",
    "PREFETCH_BUFFER_SIZE = tf.data.AUTOTUNE\n",
    "\n",
    "train_dataset_final = (train_dataset_scaled\n",
    "                       .cache()\n",
    "                       .shuffle(SHUFFLE_BUFFER_SIZE)\n",
    "                       .prefetch(PREFETCH_BUFFER_SIZE)\n",
    "                       )\n",
    "\n",
    "validation_dataset_final = (validation_dataset_scaled\n",
    "                            .cache()\n",
    "                            .prefetch(PREFETCH_BUFFER_SIZE)\n",
    "                            )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jJ2I1nk398e5"
   },
   "source": [
    "## Prepare the Model for Training\n",
    "\n",
    "You will also add data augmentation layers to avoid your model from overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6UnVzMxijYWu"
   },
   "outputs": [],
   "source": [
    "# Create a model with data augmentation layers\n",
    "data_augmentation = tf.keras.Sequential([\n",
    "    tf.keras.layers.RandomFlip(\"horizontal\"),\n",
    "    tf.keras.layers.RandomRotation(0.4),\n",
    "    tf.keras.layers.RandomTranslation(0.2,0.2),\n",
    "    tf.keras.layers.RandomContrast(0.4),\n",
    "    tf.keras.layers.RandomZoom(0.2),\n",
    "    ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uCGopMv-m3Zb"
   },
   "outputs": [],
   "source": [
    "# Attach the data augmentation model to the base model\n",
    "inputs = tf.keras.Input(shape=(150, 150, 3))\n",
    "x = data_augmentation(inputs)\n",
    "x = model(x)\n",
    "\n",
    "model_with_aug = tf.keras.Model(inputs, x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WGf3b-fK-tOl"
   },
   "source": [
    "Finally, you will compile the model with the training parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "A9ofMGh5kXMW"
   },
   "outputs": [],
   "source": [
    "# Set the training parameters\n",
    "model_with_aug.compile(\n",
    "    optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.0001),\n",
    "    loss = 'binary_crossentropy',\n",
    "    metrics = ['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3m3S6AZb7h-B"
   },
   "source": [
    "## Train the model\n",
    "\n",
    "With that, you can now train the model. You will do 20 epochs and plot the results afterwards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Blhq2MAUeyGA"
   },
   "outputs": [],
   "source": [
    "EPOCHS = 20\n",
    "\n",
    "# Train the model.\n",
    "history = model_with_aug.fit(\n",
    "    train_dataset_final,\n",
    "    validation_data = validation_dataset_final,\n",
    "    epochs = EPOCHS,\n",
    "    verbose = 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RwcB2bPj7lIx"
   },
   "source": [
    "## Evaluate the results\n",
    "\n",
    "You will use the same code to plot the results. As you can see, the validation accuracy is steady at around 95% even as your training accuracy improves. This is a good sign that your model is not overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "C2Fp6Se9rKuL"
   },
   "outputs": [],
   "source": [
    "def plot_loss_acc(history):\n",
    "    '''Plots the training and validation loss and accuracy from a history object'''\n",
    "    acc = history.history['accuracy']\n",
    "    val_acc = history.history['val_accuracy']\n",
    "    loss = history.history['loss']\n",
    "    val_loss = history.history['val_loss']\n",
    "    \n",
    "    epochs = range(len(acc))\n",
    "    \n",
    "    fig, ax = plt.subplots(1,2, figsize=(12, 6))\n",
    "    ax[0].plot(epochs, acc, 'bo', label='Training accuracy')\n",
    "    ax[0].plot(epochs, val_acc, 'b', label='Validation accuracy')\n",
    "    ax[0].set_title('Training and validation accuracy')\n",
    "    ax[0].set_xlabel('epochs')\n",
    "    ax[0].set_ylabel('accuracy')\n",
    "    ax[0].legend()\n",
    "    \n",
    "    ax[1].plot(epochs, loss, 'bo', label='Training Loss')\n",
    "    ax[1].plot(epochs, val_loss, 'b', label='Validation Loss')\n",
    "    ax[1].set_title('Training and validation loss')\n",
    "    ax[1].set_xlabel('epochs')\n",
    "    ax[1].set_ylabel('loss')\n",
    "    ax[1].legend()\n",
    "    \n",
    "    plt.show()\n",
    "\n",
    "plot_loss_acc(history)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0rc1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
